Reproducing the reported numbers

This notebook recomputes, from the data files supplied with it, every task-layer number reported in the article, and shows which line of the article each number belongs to. It computes nothing on its own: it calls independent_audit.py, the engine that produced the published values, and prints what the engine returns.

How to read it. The saved outputs below are the actual values; nothing has to be run to check them against the manuscript. To re-execute, run the cells in order from the top, or run the two scripts directly:

python independent_audit.py            # full tables, printed
python scripts/test_normalization.py   # regression tests

The engine and the tests use only the Python standard library (3.11 or later). The figure scripts additionally require numpy and matplotlib and are not run here.

What the engine reads. Four inputs, all by relative path: the de-identified task log, the reference reading of the four experimental fragments, the official JSesh Manuel de Codage correspondence table, and an independent published translation used as a second reference for the translation layer. It reads no author-derived intermediate file.

Who is in the analysis. The task log covers the participants who completed the questionnaire instrument under written informed consent. Two volunteers who submitted task work without returning a signed consent form are excluded from every number below.

code

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath("__file__")) or ".")
import independent_audit as ia

refs = ia.load_gold()
lichtheim = ia.load_lichtheim()
rows = ia.load_rows()

print(f"attempts in the log:      {len(rows)}")
print(f"participants (clusters):  {len({r['person'] for r in rows})}")
print(f"fragments with a gold reference: {sorted(refs)}")
print(f"Python: {sys.version.split()[0]}")

output

attempts in the log:      18
participants (clusters):  7
fragments with a gold reference: ['F1', 'F2', 'F3', 'F4']
Python: 3.14.6

1. Strata

Two strata are used throughout the article. The valid stratum excludes an attempt only if the response matches a fragment other than the one assigned, on both the sign and the transliteration layer, by a margin of at least 0.15. The completed stratum additionally requires coverage of at least 0.80 on both layers and a translation of at least ten tokens.

One attempt (P4, AI condition) was logged against Fragment 2 but is Fragment 1 by content; the reassignment is applied before any metric is computed, and the attempt stays in the analysis. It is flagged per row in Supplementary Data 1.

code

metrics = {}
for row in rows:
    aid = row["attempt_id"]
    metrics[aid] = ia.attempt_metrics(row, refs, lichtheim)
    closest = {L: ia.closest_fragment(row, refs, L) for L in ("signs", "translit")}
    sb, sbs, sl, _ = closest["signs"]
    tb, tbs, tl, _ = closest["translit"]
    wrong = sb == tb != row["fragment"] and sbs - sl >= 0.15 and tbs - tl >= 0.15
    metrics[aid]["wrong_fragment"] = wrong
    placeholder = "invalid because" in row[ia.F_TRANSLATION].casefold()
    metrics[aid]["complete"] = (
        metrics[aid]["signs"]["coverage"] >= 0.80
        and metrics[aid]["translit"]["coverage"] >= 0.80
        and len(metrics[aid]["tokens"]["translation"]) >= 10
        and not placeholder
        and not wrong
    )

valid = [r for r in rows if not metrics[r["attempt_id"]]["wrong_fragment"]]
completed = [r for r in valid if metrics[r["attempt_id"]]["complete"]]

print(f"valid stratum:      {len(valid)} attempts from {len({r['person'] for r in valid})} participants")
print(f"completed stratum:  {len(completed)} attempts from {len({r['person'] for r in completed})} participants")
print(f"  of which AI-assisted: {sum(r['mode'] == 'AI' for r in completed)}")
print(f"  of which unaided:     {sum(r['mode'] == 'manual' for r in completed)}")
print()
print("fragment reassignment applied to:",
      [r["attempt_id"] for r in rows if r["fragment_corrected"]])

output

valid stratum:      18 attempts from 7 participants
completed stratum:  15 attempts from 7 participants
  of which AI-assisted: 10
  of which unaided:     5

fragment reassignment applied to: [14]

2. Sign layer

Exact agreement with the reference reading, averaged over attempts carried to completion. This is the headline comparison of the Results section.

Responses entered in phonetic Manuel de Codage are resolved to canonical Gardiner codes through the official JSesh table before comparison, so that notation choice does not by itself count as disagreement.

code

def mean_precision(layer, mode, stratum):
    vals = [metrics[r["attempt_id"]][layer]["precision"] for r in stratum if r["mode"] == mode]
    return 100 * sum(vals) / len(vals)


for layer, label in (("signs", "sign layer"), ("translit", "transliteration layer")):
    ai = mean_precision(layer, "AI", completed)
    mn = mean_precision(layer, "manual", completed)
    print(f"{label:24} AI {ai:5.1f}%   unaided {mn:5.1f}%   difference {ai - mn:+5.1f} pp")

output

sign layer               AI  83.2%   unaided  85.1%   difference  -1.9 pp
transliteration layer    AI  60.2%   unaided  43.2%   difference +17.0 pp

The two values on the sign layer, 83.2 % against 85.1 %, are the ones quoted in Results; the transliteration values are 60.2 % against 43.2 %.

A note on the transliteration figure, stated here rather than left to be found: for a purely ASCII transliteration token whose initial capital letter coincides with a phonetic Manuel de Codage code, the notation system cannot be determined from the token alone — an initial T may be a Unicode t in a proper name or the MdC code for ṯ. Proper names are the only class affected in this corpus.

3. Interval around the difference

Because participants contributed unequal numbers of attempts, the interval is a participant-cluster bootstrap: participants, not attempts, are resampled, and the two sessions of one participant remain a single cluster. The engine uses 20 000 replicates with a fixed seed, so the interval below is reproducible exactly.

code

for layer, measure in (("signs", "precision"), ("translit", "precision")):
    obs, lo, hi, n_att, n_ppl = ia.bootstrap_mode_difference(
        rows, metrics, layer, measure, complete_only=True
    )
    print(f"{layer:9} AI - unaided  {obs:+.4f}   95% CI [{lo:+.4f}, {hi:+.4f}]"
          f"   {n_att} attempts / {n_ppl} participants")

output

signs     AI - unaided  -0.0187   95% CI [-0.0958, +0.0413]   15 attempts / 7 participants
translit  AI - unaided  +0.1697   95% CI [+0.1020, +0.2356]   15 attempts / 7 participants

The sign-layer difference is −0.0187, 95 % CI [−0.0958, +0.0413], over 15 attempts from 7 participants. The interval covers zero: on this evidence the two conditions are not separated on the sign layer, which is how the article reports it. The transliteration layer shows a positive difference of +0.1697.

4. Text complexity profile

The four passages were scored against a fixed corpus baseline rather than against participants' behaviour: the versioned TLA corpus of Earlier Egyptian (v18, 12 773 clauses). This reproduces the complexity table of the article.

The two corpus aggregates needed for the calculation are shipped with the package, so the table can be reproduced without obtaining the corpus. They are derived from openly licensed dumps; see ATTRIBUTION.md.

code

import subprocess

out = subprocess.run(
    [sys.executable, os.path.join("complexity", "final_metrics.py")],
    capture_output=True, text=True, encoding="utf-8",
)
print(out.stdout.strip())

output

clause distribution read from clause_logfreq_v18.json: 12773 clauses
metric                          F1      F2      F3      F4
----------------------------------------------------------
words                           49      45      34      38
signs (Gardiner)               130     118     109     113
signs/word                    2.65    2.62    3.21    2.97
finite verbs                     5       4       1       1
subord verb-forms                0       2       3       4
subord/finite                  0.0     0.5     3.0     4.0
names+titles                     5      10       4       6
var/sic flags                    6       3       3       1
content words                   22      18      19      22
% content rare(<=3)            9.1    11.1    21.1     9.1
% content absent(v18)          0.0     0.0    10.5     4.5
median content freq           24.0    68.5      24    80.0
mean log10(freq+1)            1.44    1.72    1.39    1.69
repr. percentile(v18)         50.4    60.8    33.3    37.4

rare content words (hand-verifiable, head-fallback applied):
  F1: ḥkn(2), sḏr.t(1)
  F2: sḫnti̯(3), ẖꜣm.t-jḫ.t(1)
  F3: mꜥḥꜥ.t(3), msḫn.t(0), hmhm.t(0), jw.tjt(2)
  F4: tp.j-tꜣ(0), pr.t-ꜥꜣ.t(2)

wrote frag_metrics.json

Fragment 3 is the lexically hardest and the most orthographically dense (21.1 % of its content lemmas are rare, two are absent from the corpus altogether), while Fragment 4 combines comparatively common vocabulary with the deepest syntactic embedding (4.0 subordinate forms per finite verb). The percentiles — 50, 61, 33 and 37 — are the values quoted in the text.

5. Regression tests

The tests fix the parsers, the normalisation rules, the fragment reassignment and the aggregate values reported in Results. A change anywhere in the engine that moved a published number would fail here.

code

out = subprocess.run(
    [sys.executable, os.path.join("scripts", "test_normalization.py")],
    capture_output=True, text=True, encoding="utf-8",
)
print(out.stdout.strip() or out.stderr.strip())

output

test_frozen_stratum_aggregates: OK
test_p4_correction: OK
test_population: OK
test_sign_parser: OK
test_translit_normalizer: OK
ALL TESTS PASSED

6. What this notebook does not cover